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While measures of research quality are widely accepted in the education research 
community, there may be less agreement on what constitutes evidence of impact and on 
where to look for it. The aims of this symposium are to consider some key issues in 
undertaking the Australian government’s national assessment of research engagement and 
impact, and to propose some approaches to evidencing engagement and impact in the 
context of mathematics education research. Each of the four symposium papers draws on 
our Numeracy Across the Curriculum (NAC) research program in order to ground our 
discussion in specific cases of research that have been reported at previous MERGA 
conferences. 

In the first paper, Evidencing research engagement and impact, Merrilyn Goos 
establishes the theoretical and policy context for the symposium in terms of the apparent 
lack of connection between educational research and practice. She analyses aspects of the 
NAC research program to trace rich connections between her own teaching, research and 
service roles that led to beneficial knowledge exchanges (engagement), and intricate links 
between research activities, outputs and outcomes across multiple projects (impact). Such 
an analysis can suggest “where to look” for evidence of engagement and impact. 

In the second paper, The convoluted nature of a research impact pathway, Vince 
Geiger develops a case study of an aspect of his own research within the NAC program to 
illustrate the complexity of the journey from research origin through to potential impact. 
The documentation of this research progress allows for reflection on how future impact can 
be “read” while research is taking place. 

In the third paper, Engagement and impact through research participation and 
resource development, Anne Bennison and Shelley Dole illustrate how knowledge 
exchange and uptake of resources developed through research can provide evidence of 
research engagement and impact, respectively. The analysis suggests ways in which 
collaborative research (an ARC Linkage Project on proportional reasoning and numeracy) 
and contract research (funded by the Queensland College of Teachers) can be translated for 
economic and social benefits. 

In the fourth paper, “Numeracy for learners and teachers”: Impact on MTeach 
students, Helen Forgasz evaluates the impact of a compulsory unit taken by all primary and 
secondary pre-service teachers in the Monash University Master of Teaching. The unit 
design incorporates elements of the Numeracy Across the Curriculum model to address 
AITSL standards for knowledge and understanding of literacy and numeracy teaching 
strategies, and interpreting student data. The evaluation reveals substantial impact on 
students’ understanding of numeracy and confidence in incorporating numeracy in their 
teaching, thus highlighting the contribution of research to improving teacher education. 


(2017). In A. Downton, S. Livy, & J. Hall (Eds.), 40 years on: We are still learning! Proceedings of the 40th 
Annual Conference of the Mathematics Education Research Group of Australasia (p. 629). Melbourne: 
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This paper outlines current developments in the Australian government’s plan to introduce 
a national assessment of research engagement and impact and considers implications for 
mathematics education. A well-established research program seeking to embed numeracy 
across the school curriculum is used to illustrate forms of research engagement and impact. 
The analysis of this program demonstrates rich connections between research, policy and 
practice, and suggests “where to look” for evidence of engagement and impact. 


In December 2015, as part of its National Innovation and Science Agenda, the 
Australian government announced the development of a national assessment of research 
engagement and impact. It is envisaged that the assessment will be implemented in parallel 
with the national evaluation of research quality — Excellence in Research for Australia 
(ERA). The Australian Research Council (ARC) and the Department of Education and 
Training released an Engagement and Impact Assessment Consultation Paper in May 2016 
to seek feedback from stakeholders on how this assessment should be undertaken (ARC 
and DET, 2016). 

While measures of research quality are widely accepted in the education research 
community, there may be less agreement on what constitutes evidence of impact and on 
where to look for this evidence. The aims of this paper are to consider some of the key 
issues in undertaking a national assessment of research engagement and impact that were 
raised in the Consultation Paper, and to propose some approaches to evidencing 
engagement and impact in the context of mathematics education research. 


Theoretical Background: The Gap between Research, Policy, and Practice 


Education research is often criticised for its lack of impact on classroom practice. 
Explanations for the apparent research-practice gap sometimes highlight the different 
processes used by researchers and teachers to improve educational practice. For example, 
Richardson (1994) suggested that, whereas formal research aims to contribute to an 
established and general knowledge base, the practical inquiry of teachers is focused on 
solving immediate day-to-day problems. Writing from an educational leadership and policy 
perspective, Levin (2010) invoked the idea of knowledge mobilisation to examine 
connections between the production, communication, and use of research. He argued that 
not only do researchers have a responsibility communicate their findings beyond academia, 
but policy-makers and practitioners also need to be willing to find, share, and use good 
research in their work. 

The apparent lack of connection between education research, policy and practice seems 
to be particularly relevant to mathematics education. For example, national and 
international assessments of mathematics achievement, such as the National Assessment 
Program — Literacy and Numeracy (NAPLAN), the OECD’s Programme for International 
Student Assessment (PISA), and the IEA’s Trends in International Mathematics and 
Science Study (TIMSS) allow governments to compare performances within and between 
countries and can create pressure to change mathematics curricula and teaching practices. 
Nevertheless, it is well documented that classroom practice remains resistant to the reform 
approaches promoted by mathematics education researchers (e.g., Gill & Boote, 2012). In 
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light of discussions in the literature and the move within Australia towards a national 
assessment of research engagement and impact, it is especially timely for mathematics 
educators to consider how to evidence the uptake and benefit of their research. 


Defining and Evidencing Research Engagement and Impact 


For the purpose of illustration, I refer to the Numeracy Across the Curriculum (NAC) 
research program to which the presenters of this symposium have contributed in different 
combinations and in different ways. The program builds on sixteen years of productive 
engagement with teachers, teacher educators, policy-makers, school systems, and the 
Australian and international research community. The research was motivated by a desire 
to challenge narrow “basic skills” interpretations of numeracy that prepare low-achieving 
students to do no more than “survive” in the world beyond school. As a result, the research 
team developed a rich interpretation of numeracy that connects the mathematics learned at 
school with out-of-school situations that additionally require problem solving, critical 
judgment, and making sense of the non-mathematical context. This approach necessarily 
positions numeracy as an across-the-curriculum commitment that extends beyond the 
mathematics classroom. The most significant outcome of the program is a model of 
numeracy for the 21st century that recognises the intellectual, affective, and contextual 
demands of becoming a numerate person. 

According to this model, numeracy development requires attention to real-life contexts, 
the application of mathematical knowledge, the use of representational, physical and digital 
tools, and positive dispositions towards the use of mathematics. A further important and 
overarching element of the model is a critical orientation to the use of mathematics (see 
Goos, Geiger, & Dole, 2014). 


Research Engagement 


The ARC Consultation Paper draws on the definition used by the Academy of 
Technological Sciences and Engineering (ATSE) to develop metrics for Australian 
universities’ research engagement. Engagement was defined as: 

the interaction between researchers and research organisations and their larger communities/ 


industries for the mutually beneficial exchange of knowledge, understanding and resources in a 
context of partnership and reciprocity. (ATSE, 2015) 


However, the Consultation Paper notes that metrics, which are largely based on research 
commercialisation income and patents, may not capture the complexity of some forms of 
research engagement. As a qualitative alternative, Figure 1 maps the interactions between 
my own academic teaching, research, and service roles that led to beneficial knowledge 
exchanges in teacher education, professional development, and consultancy settings, 
involving practitioners, school leaders, education systems, professional associations, and 
teacher registration authorities as part of my contribution to the NAC research program. 
Also noticeable in this diagram is the absence of a neat linear progression from research 
contexts towards contexts of application. Instead, knowledge exchange has built reciprocal 
relationships across the boundaries between research, policy, and practice. 


Research Impact 


The ARC (2012) defines research impact as “the demonstrable contribution that 
research makes to the economy, society, culture, national security, public policy or 
services, health, the environment, or quality of life, beyond contributions to academia”. 


Teaching Service Research 


2000-2002 Curriculum integration 2002-2003 DEST project: Home, 
project (initial teacher education) school, community partnerships to 
support numeracy 


2005 Qld Board of Teacher 
Registration: develop numeracy 
standards for graduate teachers 
2007 SA Dept. Education & —_—_—_—>)} Rich model 
Children’s Services (DECS) of numeracy 
consultancy on Numeracy in the 
FutureSACE 


(1) 2009 SA DECS contract 
research 


(2) 2010-11 Brisbane Catholic 
Education (BCE) contract research 


(3) 2012 BCE contract research 
(4) 2012-14 ARC Discovery 
Project 
(5) 2010-14 ARC Linkage Project 
(6) 2012-16 PhD project 

2015 Masters course on literacy & (7) 2014-15 Qld College of 


numeracy across the curriculum Teachers web-based resource 
2017 ITE course on numeracy development 
across the curriculum (8) 2010-12 AAMT Make It 


Count numeracy evaluation 


Figure I. Personal engagement map: Numeracy across the curriculum research program 


While noting that there are no clearly defined indicators for research impact, the 
Consultation Paper refers to peer reviewed case studies — conducted as part of the recent 
UK REF exercise — as being an appropriate means of assessment. Nevertheless, case 
studies are expensive to produce. The ARC has also developed a Research Impact Pathway 
table (http://www.arc.gov.au/research-impact-principles-and-framework#table) to assist 
grant applicants to identify the potential benefits of their proposed research. The table’s 
column headings are listed below, together with education-relevant examples: 


Inputs: budget, research assistants, infrastructure; 

Activities: research project, undergraduate teaching, professional development; 
Outputs: publications, PhD graduates, resources developed; 

Outcomes: uptake of resources, research incorporated into teacher education and 
support materials, changes in policy based on research evidence; 

5. Benefits: improved outcomes for learners, improved teaching practice. 
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Figure 2 provides a mapping of some of the outputs and outcomes of the NAC research 
program, using the same eight research activities identified in Figure 1. Evidencing direct 
benefits to students and teachers remains a pressing challenge for much education research. 


Sample Research Outputs and Outcomes eciied cars 

#1 #2 #3 #4 #5 #6 #7 #8 
Curriculum audit methodology * ** 
Principles for task design and curriculum planning 7 ** * * * 
Professional development approach * * * * * * * 
Methods for monitoring teacher learning trajectories *k ** ** 
Whole school approaches to numeracy leadership * * * * 
Resources for teachers and teacher educators * * * ** * 


Figure 2. Partial impact map: Numeracy across the curriculum research program. 


Issues for Consideration 


Two of the key issues identified in the Consultation Paper are worth considering here. 
The first involves accounting for the variable time lags between undertaking research and 
achieving identifiable benefits for end-users. As Figure 1 shows, it can take more than ten 
years for education research to make a demonstrable contribution to society. The second 
issue refers to the difficulties in determining the attribution of research engagement and 
impact, for example, if an impact can be traced back to more than one project, as is the 
case in the NAC research program (see Figure 2). Not only was impact derived across 
multiple research activities, but these activities also spanned the multiple universities in 
which the research team members worked. It remains to be seen how a national assessment 
of impact could be undertaken if the unit of analysis is the individual university. 

Beyond the immediacy of an impending national assessment of research engagement 
and impact, there is surely value for mathematics educators in retrospectively analysing our 
own research to illuminate the opportunities taken, decisions made, and relationships built 
in pursuing research that makes a difference. Such an analysis might help us not only to 
learn “where to look” for evidence of past impact, but also to plan future research projects 
with an eye to demonstrating potential benefits for educational policy and practice. 
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How research inputs and activities translate into outputs, outcomes and benefits is an 
increasingly important question within Australian mathematics education research. The 
pathway from the development of new ideas that drive educational projects through to 
innovations that have broad influence at local, national and international levels, however, is 
often convoluted and notoriously difficult to strategise. In this paper, I develop a case study 
of an aspect of my own research to illustrate the complexity of the journey from research 
origin through to impact. The documentation of this research progress allows for reflection 
on how future impact can be “read” while research is taking place. 


The generation of innovative ideas and procurement of funds for testing new 
approaches in the field is at the heart of research in education. However, the “impact” of 
such research, educational outcomes and benefits to society, is no longer seen as a matter 
of potential but rather an expectation by increasingly numbers of funding bodies. While 
measures of research quality have been a topic of discussion for more than a decade within 
the Australian research context, the Engagement and Impact pilot currently being 
conducted by the Australian Research Council (ARC) provides evidence that a sharper 
focus will be drawn on this issue in future funding rounds. The proposed Engagement and 
Impact Assessment, as part of the National Innovation and Science Agenda (Australian 
Government, 2017) will consider: research interactions with a broad range of stakeholders 
including: industry; Government, non-governmental organisations; and research 
contributions to the economy, society and environment. This assessment will be conducted 
as a companion exercise to the Excellence in Research for Australia and is anticipated to be 
a significant consideration in future ARC application assessments. Such an exercise poses 
the challenge of how researchers will identify and then document the impact of their work? 
And, perhaps more importantly, dares researchers to think if it is possible to consider or 
strategise impact — to “read” impact — as a component of the research enterprise before 
and/or during the conduct of an investigation and not just in hindsight. 

In this paper, I present a case study based on my contribution to the Numeracy Across 
the Curriculum (NAC) research program to illustrate the complexity of the journey from 
research origin through to impact. The journey began with a small research project and was 
sustained via a series of both self-generated and serendipitous opportunities. Analysis of 
this journey will focus on how perceptions of impact can be constructed through hindsight. 
A brief reflection on implications for “reading” impact in order to inform decisions about 
future individual, team or institution research behaviour will conclude the paper. 


Research Impact Pathway 


As a feature of the advice provided by the ARC on the nature of research impact a 
Research Impact Pathway (RIP) table (Australian Research Council, 2017) was developed. 
The Pathway depicts impact as a progression through five junctures — Inputs, Activities, 
Outputs, Outcomes and Benefits. Forms of evidence for each juncture are exemplified 
within the table, for example, possible Inputs include research income, staff, background 
IP, infrastructure and collections, while the exemplars of Outcomes are listed as 
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commercial products, licences and revenue, new companies — spin offs, start ups or joint 
ventures, job creation, implementation of programs and policy, citations and integration 
into policy. Progression towards impact is presented in a sequential, linear fashion as if one 
step leads naturally to the next. What happens in practice however is likely to be more 
convoluted. The convoluted nature of my impact progression within the NAC program is 
presented in the remainder of this paper. 


Method 


Figure 1, complemented by Tables 1-3, is a representation of evidence within impact 
junctures, defined by the RIP table, against time. The evidence included here relates to 
aspects in which I have been directly involved within the NAC research program, for 
example, authorship or co-authorship of publications, as named investigator or co- 
investigator of a project. Inputs are in the form of research income associated with projects 
(Table 1) supported via a number of funding sources. Only Activities (Table 2) that have 
direct connection to the NAC research program have been included. Outputs consist of 
publications and teaching resources. Outcomes have been identified as results of research 
that have receive attention across education systems or internationally. To date, I don’t 
believe the program can have serious claim to Benefits as exemplified in the RIP table. 
Thus, Benefits is represented as a blank rectangle in Figure 1. 
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Figure I. Representation of Geiger's RIP through the NAC research program. 
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Solid black arrows have been used to indicate publications or resources that were a 
direct Output from research projects (Inputs). The number of Outputs connected to an 
Input is recorded next to an arrow when this exceeds one. 


Table 1 
Inputs: Research Income 


Project Time Project name 

A 2009 Numeracy in the learning areas (middle years) 

B 2010-2011 Leading numeracy learning 

C 2010-2012 Make it count: Numeracy, mathematics and Indigenous learners 

D 2012 Sustaining numeracy curriculum leadership: A whole school approach 

E 2012 Models of leading curriculum reform in numeracy 

F 2012-2014 Enhancing numeracy learning and teaching across the curriculum 

G 2014-2015 Numeracy teaching across the curriculum in Queensland: Resources for 
teachers 

H 2015-2017 Designing and implementing cross-curricular numeracy tasks for 
effective teaching and learning 

I 2016 Review of the PIAAC numeracy assessment framework 

Table 2 

Activities 

Activity Time Activity name 

J 2016 ICME Topic Study Group Plenary 

K 2014-2015 Guest Editor ZDM (Special Issue — Numeracy) 

Table 3 

Outcomes 


Activity Time Activity name 


L 2009 Organising structure for Numeracy in the Middle Years Curriculum — 
Department of Education and Children’s Services, South Australia 


2011- An instructional planning tool adopted by Brisbane Catholic Education 
2015 Numeracy skills framework — Department of Education NSW 

2011- Numeracy teaching resource package on Education Queensland’s website 
2015- Numeracy across the curriculum resource package on QCT Website 


ORS Ze 


2016 One of Springer’s most downloaded chapters in the last two years and was 
made freely available as a part of their World Teacher’s Day promotion. 


R 2011- Citations (177) 


Dashed arrows have been used to indicate outcomes from any of the preceding 
junctures — Inputs, Activities and Outputs. For example, the Outcome, Numeracy skills 
framework, developed by the Department of Education NSW draws directly on the NAC 
team’s research to provide system wide advice to teachers about the integration of 
numeracy across the curriculum. In contrast, the dotted arrows flow in the opposite 
direction of the assumed RIP illustrating how research Outputs can also feedback into 
Activities that eventually new projects (Inputs). The specific example identified in Figure 1 
relates to how publication Outputs helped build a case for a special issue of ZDM — 
Mathematics Education on Numeracy (2015) (Activities). This issue, resulted in a personal 


invitation for me to be part of the organising committee for the Topic Study Group (TSG) 
on Mathematical Literacy at ICME 2016 (Activity) and an additional invitation to deliver a 
plenary with two NAC team members (Goos and Forgasz) within the TSG 
(Activities/Outputs). An invitation from the OECD for me to contribute to Review of the 
PIAAC numeracy assessment framework managed by ACER (Input) followed from this 
series of events. This new Input will be the foundation of additional outputs and activities. 


Conclusions 


The general, direction of flow from Inputs to Outputs to Outcomes in Figure 1 is 
consistent with that of the RIP table, shown with solid and dashed arrows. The role of 
Activities, however, seems to be absent in the actualised pathway of my contribution to the 
NAC program — giving the appearance that Activities play no role in moving from Inputs 
to Outcomes. Perhaps this is because the exemplars provided in the RIP table are too 
limited (research work and training, workshop/conference organising, facility use, 
membership of learned societies and academies, and community and_ stakeholder 
engagement), restricting my selection of Activities and so further consideration is needed 
for what happenings can be considered Activities. Journal articles, for example, are not a 
direct output from research income (Inputs) as first the research itself must be conducted, 
data gathered and analysed, the article written (and usually revised); all of which are more 
akin to Activities than Inputs. Additionally, some influences may be too subtle to be 
captured in a representation such as Figure 1, such as promoting the need for research on a 
particular issue through a Learned Society — which leads to funds from concerned parties 
becoming available for research; a circuitous but still productive pathway. 

The pathway indicated by the dotted arrows, however, flows in the opposite direction 
of that of the RIP table and demonstrates that RIP junctures can be bi-directionally 
influential; in this case moving from Outputs back to Inputs. Activities, in this pathway 
(Table 2), were significantly influential in highlighting the quality of the work in the NAC 
research program, leading to the inclusion of a NAC team member in a new project of 
international standing — a Review of the PIAAC numeracy assessment framework (Inputs). 

The preceding analysis shows that my contribution to the NAC research program can 
be mapped from inputs to outcomes via the RIP sequence but also demonstrates that 
Outputs, at least, can be backward mapped to new research endeavours (Inputs). The trends 
identified here leads to questions about individual and institutional behaviours related to 
impact. How would an analysis of impact differ when considering an individual, research 
team or institution and what strategic decisions would result at each level? Is it possible to 
utilise trends identified via RIP sequences using hindsight to make strategic decisions 
about the type Activities, Outputs or Outcomes an individual, research team or institution 
should pursue in order to make greatest impact and/or lead to further research? What 
measures can be employed in order to shorten the timeframe of the RIP sequence — adding 
substance to claims of successful research investment? These are questions that will need 
to be addressed as we move into the new era of Engagement and Impact. 
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This paper utilises two projects that are part of a well-established research program seeking 
to embed numeracy across the curriculum to illustrate how knowledge exchange and uptake 
of resources can be used to provide evidence of research engagement and impact. The aim 
of the first project was to build teachers’ pedagogy around the promotion of proportional 
reasoning as a cross-curricular concept and a key component of numeracy; whilst that of 
the second was to develop resources to assist teachers to embed numeracy across the 
curriculum. Participation of stakeholders and the resources produced provide evidence of 
engagement and impact of the two projects. 


A national assessment of research engagement and impact that requires universities to 
provide evidence of how research is translated into economic, social, and other benefits is 
part of the Australian Government’s National Innovation and Science Agenda. An 
Engagement and Impact Assessment Consultation Paper, released in May 2016, sought 
feedback from stakeholders on how this assessment should be undertaken (ARC and DET, 
2016). Several key issues associated with measuring engagement and assessing impact 
were raised in the Consultation Paper. The aim of this paper is to illustrate how knowledge 
exchange and the uptake of resources by teachers and education systems could be used to 
evidence research engagement and impact. 


Research Engagement and Impact 


Research engagement has been defined as the exchange of knowledge, understanding 
and resources that result from interactions between researchers and their wider 
communities (ATSE, 2015). The emphasis is on research benefit. In a recent review of 
trends and strategies for commercialising public research, the OECD (2013) noted that 
there are multiple ways in which research can be translated for economic and social 
benefits: 


Knowledge transfer and commercialisation of public research refer in a broader sense to the 
multiple ways in which knowledge from universities and public research institutions (PRIs) can be 
exploited by firms and researchers themselves so as to generate economic and social value and 
industrial development. (p. 18) 


Among the various forms of research engagement considered of high significance for 
industry, and seen therefore as more directly transferable (and hence more impactful) are 
collaborative research and contract research (ARC and DET, 2016). Both these types of 
research were undertaken as part of the Numeracy Across the Curriculum (NAC) research 
program undertaken by the presenters of this symposium. 

Impact is more difficult to define and assess. The ARC’s Research Impact Pathway 
table (http://www.arc.gov.au/research-impact-principles-and-framework#table) provides 
one way of identifying potential benefits of proposed research. The table includes 
examples of impact at five stages over the life of a research project and beyond its formal 
conclusion under the headings of Inputs, Activities, Outputs, Outcomes, and Benefits. 
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In this paper, we draw upon two specific projects from the NAC research program in 
an attempt to illustrate an approach to evidencing engagement and impact. The projects (an 
ARC Linkage and an Industry funded project) fit the categories of collaborative and 
contract research respectively, thus by their nature evidence research engagement. We 
analyse the impacts of these two projects in relation to inputs, activities, outputs, outcomes, 
and potential benefits to further evidence engagement and highlight the impact of each. 
Both these projects drew on the model of numeracy for the 21st century developed earlier 
in the NAC research program. According to this model, numeracy development 
encompasses five dimensions: mathematical knowledge, context, tools (representational, 
physical and digital), and positive dispositions toward the use of mathematics, which are 
embedded in a critical orientation (Goos, Geiger, & Dole, 2014). 


ARC Linkage Project 


The Enhancing Proportional Reasoning project was an ARC Linkage project 
conducted in 2010-2014 and included collaborative partner funding of 30%. The study 
found that improving teachers’ understanding of the elements of the numeracy model 
broadened their teaching focus beyond the teaching of mathematical knowledge. Teachers 
became more aware of the need to incorporate tools, including digital tools to enhance 
students’ numeracy ability. They more directly included a focus on student dispositions 
(e.g., confidence, resilience and risk taking) as a key element of promoting personal 
numeracy. Teachers also reported a greater awareness of and ability to identify “numeracy 
moments” in cross-curricular circumstances, thus broadening students’ numeracy 
development opportunities and making this acquisition more “real life” (see Dole, Hilton, 
G., & Hilton, A., 2015). Teachers indicated that regular identification of proportional 
reasoning teaching and learning opportunities in cross-curricular contexts led to students’ 
improved ability to identify and work with proportional situations as well as improving 
their meta-language, allowing them to communicate their ideas about proportional 
situations more precisely and concisely (see Hilton, A., Hilton, G., Dole, & Goos, 2016). 
This project has received international recognition for its research/practitioner focus 
(Hilton, A., Hilton, G., Dole, & Goos, 2013). 


Industry Project 


Numeracy teaching across the curriculum in Queensland: Resources for teachers was 
an Industry project conducted in 2014-2015 in response to a call from the Queensland 
College of Teachers (QCT). The project addressed a particular need of the QCT: to 
enhance the teaching of numeracy across the curriculum through web-based resources that 
could be made readily available to teachers via the QCT website. The project included a 
literature review of national and international good practice, an audit of existing material 
and consultation with stakeholders (e.g., employing authorities and teacher professional 
associations) to identify gaps and areas where teachers would benefit from new resources, 
developing video vignettes of examples of good practice in Queensland schools, and 
providing a brief report to the QCT (Goos, Geiger, Bennison, & Roberts, 2015). The 
theoretical framework that informed resource development encompassed the Board of 
Teacher Registration, Queensland (2005) Numeracy Standards and model of numeracy 
developed by the NAC research program. 

The audit of existing materials and interviews with stakeholders revealed that there are 
very few resources available to support teachers’ understanding and enactment of 


numeracy across the curriculum. The findings highlighted important gaps including that 
almost none of the existing materials addressed the need for teachers to develop the 
capacity to recognise and take advantage of the numeracy learning demands and 
opportunities within the subjects they teach. In response to these findings, six video 
vignettes were produced: an interview with a numeracy expert that explains some of the 
evidence base for the examples of good practice, a set of four classroom vignettes 
illustrating good practice in teaching numeracy across the curriculum at different year 
levels and in different subjects, and an interview with a school numeracy team that 
provides an example of how a whole school approach to numeracy can be developed. 


Discussion and Concluding Remarks 


It is possible to identify evidence of engagement and impact in the two projects 
presented here. There was knowledge exchange between researchers and stakeholders in 
the consultation process, publications, and availability of resources produced. The impact 
of the projects, using the column headings in the ARC’s Research Impact Pathway table, 
are summarised in Figure 1. 


ARC Research ARC Linkage Industry Project 
Impact Pathway 
Inputs Co-funding from ARC and industry Funding from the QCT 
partners (Education Authorities in two 
states) 
Activities 5 regional school clusters in two states; Collaborative stakeholder engagement 
60 classroom teachers; (teacher registration and employing 
10 school leaders; authorities, teacher professional 
PD package for teachers (10 modules); associations); 
Resource development; Resource development 
State conference on Proportional 
Reasoning in both states (105 and 130 
attendees respectively); 
Outputs 6 refereed journal articles; Brief report to QCT; 
5 refereed conference papers; 1 refereed conference paper; 
6 conference presentations; 6 video vignettes 
1 international research award; 
Book proposal 
Outcomes 11 invited keynote addresses, national Video vignettes made available on the 
and international; QCT website 
20 teacher workshop presentations; (https://www.filmpond.com/#/ponds/qct- 
Citations; the-university-of-queensland) 
Integration into school policy 
Benefits Potential for improved teaching practice Potential for improved teaching practice 
and improved outcomes for learners and improved outcomes for learners. 


Figure 1. Mapping of project impact against ARC’s Research Impact Pathway. 


Knowledge transfer between researchers and stakeholders, along with resources that 
have been taken up by stakeholders, provide evidence of engagement and impact of the 
ARC Linkage Project and the Industry Project. The outcomes and outputs of the two 
projects are summarised in Figure 2 and illustrate how each project contributes to the 
engagement and impact of the NAC research program which has been conducted over a 
16-year period by researchers in multiple universities, with outputs and outcomes being 
built upon in successive projects. 


Research Activities 
Sample Research Outputs and Outcomes 2010-2014 ARC Linkage 2014-2015 Industry 
Project Project 

Principles for task design and curriculum planning * * 
Professional development approach * * 

Whole school approaches to numeracy leadership * * 
Development of resources for teachers * * 
Assessment of numeracy capability * 


Figure 2. Impact map for ARC Linkage Project and Industry Project. 


This analysis also illustrates two issues identified in the Consultation Paper; that is, the 
time lag between research and benefits for end-users and the difficulty in attributing impact 
to a single project or university. 
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“Numeracy for learners and teachers” is a compulsory unit for all primary and secondary 
pre-service teachers in the Monash University Master of Teaching course. In the first two 
years that the unit was taught (2015 and 2016), research was conducted to evaluate the 
unit’s impact on students’ understanding of the construct, numeracy, and on their 
confidence to incorporate numeracy in their teaching across the curriculum. In both years, 
surveys were administered before commencement and on completion of the unit; a small 
number of students were also interviewed. The major findings from the two-year study are 
presented in this paper. 


Introduction 


There were two main drivers for the development of the unit, Numeracy for Learners 
and Teachers (EDF5017), as a compulsory study in the Monash University Master of 
Teaching (MTeach) course. All MTeach students, except those focusing only on becoming 
teachers in the early years, must complete this unit. The two drivers were: 
1. Numeracy as one of seven general capabilities in the Australian Curriculum (AC; 
Australian Curriculum, Assessment and Reporting Authority [ACARA], 2016), the 
basis of the curriculum in each state/territory of Australia. “Teachers are expected 
to teach and assess general capabilities to the extent that they are incorporated 
within learning area content” (ACARA, n.d.-a) In the AC, numeracy is defined as 
encompassing “the knowledge, skills, behaviours and dispositions that students 
need to use mathematics in a wide range of situations. It involves students 
recognising and understanding the role of mathematics in the world and having the 
dispositions and capacities to use mathematical knowledge and skills purposefully” 
(ACARA, n.d.-b). 
2. The graduate standards developed by the Australian Institute for Teaching and 
School Leadership (AITSL). These standards must also be met as part of the 
accreditation process for providers of teacher education. The specific AITSL 
(2014) graduate standards underpinning the development of EDF5017 were: 
¢ Standard 2.5 Literacy and numeracy strategies: “Know and understand literacy 
and numeracy teaching strategies and their application in teaching areas”. 

¢ Standard 5.4 Interpret student data: “Demonstrate the capacity to interpret 
student assessment data to evaluate student learning and modify teaching 
practice”. 

The definition of numeracy embraced in EDF5017 is that adopted in the AC and 
described above. The elements of the 21st Century Model of Numeracy (Goos, Geiger, & 
Dole, 2014) scaffolded the curriculum design for the unit. Focusing on the identification of 
numeracy demands and opportunities across all AC curricular domains at all grade levels, 
as well as developing the personal numeracy skills needed by practicing teachers were 
among the outcome goals of the unit. 
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EDF5017 was first offered in 2015. In that year, the cohorts enrolled were MTeach 
(Secondary) and MTeach (Primary/Secondary) students; in 2016, the cohorts were MTeach 
(Primary) and MTeach (Early Years/Primary) students. Due to a revision of the timing of 
some MTeach offerings, from 2017, the four cohorts of students will be enrolled in the unit 
simultaneously. 

In this paper, I present some of the findings from a two-year study conducted with the 
EDF5017 students that was aimed at evaluating the unit’s overall impact. The research 
focus reported here is on students’ understanding of the construct, numeracy, and on the 
students’ confidence to incorporate numeracy in their teaching across the curriculum. The 
MTeach cohort split in 2015 and 2016 enabled the data to be examined for any differences 
among secondary pre-service teachers (2015) and primary pre-service teachers (2016). 


Methods and Analyses 


A mixed methods approach was adopted to gather data to evaluate the success of the 
unit. Pre- and post-surveys were developed so that changes in views in response to 
studying EDF5017 could be gauged. The items focused on in this paper include those 
designed to identify changes in the pre-service teachers’ understandings of the construct 
numeracy, and in their confidence to incorporate numeracy in teaching. Interviews were 
conducted with volunteers a short time after the post-survey had been completed, the main 
aim being to gather views on the content and structure of the unit. 


Results and Discussion 


The Samples 
The pre- and post-survey samples in 2015 and 2016 are shown in Table 1. 


Table 1 
2015 and 2016 Pre- and Post-Survey Samples 


2015 2016 

Pre-survey Post-survey Pre-survey Post-survey 
Participants 53 began; 40 35 began; 20 46 began; 22 21 began; 13 

finished finished finished finished 
Gender 81% female 74% female 90% female 81% female 
Age 77% aged 25-34 74% aged 25-34 80% aged 25-34 86% aged 25-34 
MTeach Secondary only Secondary only Primary only Primary only 
stream (74%) (80%) (79%) (90%) 


As can be seen in Table 1, most respondents were female, and most were aged 25-34. 
Of the 2015 cohort, more were enrolled in MTeach (Sec) than in MTeach (Prim/Sec); for 
the 2016 cohort, more were enrolled in MTeach (Prim) than in MTeach (EY/Prim). 


Findings 


Differences between numeracy and mathematics. Responses to the item “Are there 
differences between mathematics and numeracy?” (Yes/No/Unsure) to the 2015 and 2016 
pre- and post-surveys are shown in Table 2. 


Table 2 
Are there Differences between Mathematics and Numeracy? 


2015 2016 
Pre-survey Post-survey Pre-survey Post-survey 
(n = 45) (n= 21) (n = 29) (n = 13) 
Yes 76% 95% 90% 92% 
No 4% 0% 0% 8% 
Unsure 20% 5% 10% 0% 


As can be seen in Table 2, for the 2015 cohort, there was a noteworthy increase in the 
proportion of secondary pre-service teachers who answered “Yes” after completing studies 
in EDF5017. While starting from a very high base (90% of students), among the 2016 
primary pre-service teacher cohort, there is no noticeable difference in the proportions 
saying “Yes” in the pre- and post-surveys. One possible explanation for the 2016 cohort 
being aware that there is a difference is that this cohort had already completed units in the 
teaching of primary mathematics and the issue had already been discussed in those units. 

Participants were also asked to explain their answers to the question. Typical answers 
are shown below: 


I'd never really given it much thought before now. Both scare me!!! 
One is the subject, the other is the application of the subject in real life situations. 
Mathematics is to numeracy what language is to literacy - only part of the whole. 


I think that numeracy is a broader concept than mathematics, because otherwise we wouldn’t have 
pure maths. 


Confidence incorporating numeracy into the teaching of their subject area(s). On both 
the pre- and post-surveys, respondents were asked to indicate on a 5-point response format 
(very lacking in confidence to very confident) how confident they felt about incorporating 
numeracy into the teaching in their subject area/s. The pre- and post-survey responses for 
the 2015 and 2016 samples are shown in Figures | and 2 respectively. 


It can be seen in both Figures 1 and 2, that there were noteworthy changes in 
confidence from pre- to post-survey. That is, in both samples of pre-service teachers, more 
were somewhat or very confident after studying EDF5017 than before. 


Figure I. Pre- and post-survey responses from 2015 participants. 


Percentage 


Perentage 


m@ Pre-Unit m Post-Unit 
a i ' a les | 


Verylackingin | Somewhat Neither Somewhat Very confident 
confidence lacking in confident nor confident 
confidence lacking in 
confidence 


Confidence Level 
Figure 2. Pre- and post-survey responses from 2016 participants. 


The following comment from one of the 2015 (secondary) post-survey respondents 
encapsulates the sentiments of many of the students: 
I have a clearer understanding of what numeracy entails, have been provided examples with how it 


would work in my method curriculum areas, and feel confident that I have adequate mathematical 
reasoning and numeracy skills to be able to handle this in my teaching. 


In the post-survey only, students were asked if the unit had impacted their views of 
numeracy. The majority responded, “Yes” (86% in 2015, 85% in 2016). Some typical 
explanations for their positive responses included: 


I did not know the word before this unit. 
I understand it is my responsibility to teach this [numeracy] — AITSL and curriculum require it. 


I now know the difference between mathematics and numeracy. 


Final Words 


Clearly, completing EDF5017 resulted in a substantial and important impact on 
students’ confidence in incorporating numeracy in their teaching, and in having a better 
appreciation of what numeracy is and how it differs from mathematics. Units such as 
EDF5017 are now expected for accreditation of teacher education programs. Based on the 
findings reported here, it is anticipated that the benefits to the school population and the 
future citizenry of Australia are likely. 
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